3 mos camera

ABSTRACT

A 3 MOS camera includes a first prism that causes a first image sensor to receive IR light of light from an observation part, a second prism that causes a second image sensor to receive visible light of A % (A: a predetermined real number) of the light from the observation part, a third prism that causes a third image sensor to receive remaining visible light of (100-A)% of the light from the observation part, and a video signal processor that combines a color video signal based on imaging outputs of the second image sensor and the third image sensor and an IR video signal based on an imaging output of the first image sensor and outputs the combined signal to a monitor, the second image sensor and the third image sensor being respectively bonded to positions optically shifted by substantially one pixel.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present disclosure relates to a 3 MOS camera.

2. Background Art

In recent years, attention has been paid to a diagnosis method in which,at the time of surgery or examination, ICG (indocyanine green) isadministered as a fluorescent reagent into a subject, and the ICG isexcited by emission of excitation light or the like to capture andobserve a near-infrared fluorescence image emitted by the ICG togetherwith a subject image. For example, JP-A-2016-75825 discloses an imagingdevice having a blue separation prism that reflects a part of bluecomponent light of incident light and near-infrared light in a specificwavelength region and transmits light other than the above light, a redseparation prism that reflects a part of red component light of incidentlight and near-infrared light in a specific wavelength region andtransmits light other than the above light, and a green separation prisminto which the light transmitted through the red separation prism isincident.

In a configuration in JP-A-2016-75825, a partial light amount of thenear-infrared light of light from a diseased part or the like isincident on each of the plurality of color separation prisms in a sharedmanner and imaged. For this reason, for example, there is a problem inthat light specialized in the wavelength region of the near-infraredlight cannot be received by a corresponding imaging element. Therefore,it is difficult to output a clearer fluorescence image of an observationpart to which the fluorescent reagent is administered at the time ofsurgery or examination described above, and there is room forimprovement in that a doctor or the like can more easily grasp thediseased part. Each of blue, red, and green lights is specially imaged.Therefore, there is room for improvement in enhancing resolution of avideo by imaging visible light.

SUMMARY OF THE INVENTION

The present disclosure has been devised in view of the above-mentionedcircumstances, and a purpose thereof is to provide a 3 MOS camera thatachieves both generation of a clearer fluorescence video of anobservation part to which a fluorescent reagent is administered andresolution enhancement of a color image of the observation part toassist a doctor or the like in easily grasping a diseased part.

The present disclosure provides a 3 MOS camera including a first prismthat causes a first image sensor to receive IR light of light from anobservation part, a second prism that causes a second image sensor toreceive visible light of A % (A: a predetermined real number) of thelight from the observation part, a third prism that causes a third imagesensor to receive remaining visible light of (100-A)% of the light fromthe observation part, and a video signal processor that combines a colorvideo signal based on imaging outputs of the second image sensor and thethird image sensor and an IR video signal based on an imaging output ofthe first image sensor and outputs the combined signal to a monitor, thesecond image sensor and the third image sensor being respectively bondedto positions optically shifted by substantially one pixel.

According to the present disclosure, it is possible to achieve both thegeneration of the clearer fluorescence video of the observation part towhich the fluorescent reagent is administered and the resolutionenhancement of the color image of the observation part, and thus toassist the doctor or the like in easily grasping the diseased part.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram showing an internal configuration example ofa 3 MOS camera according to a first embodiment.

FIG. 1B is a block diagram showing another internal configurationexample of the 3 MOS camera 1 according to the first embodiment.

FIG. 2 is a diagram showing a structural example of a spectral prismshown in FIG. 1.

FIG. 3A is a diagram showing an arrangement example of color filters ofimaging elements 151 and 152.

FIG. 3B is an explanatory diagram of a problem in a case where the colorfilters of the imaging elements 151 and 152 are configured in a Bayerarray and are disposed with half pixel shifting.

FIG. 4A is a graph showing an example of spectral characteristics of adichroic mirror.

FIG. 4B is a graph showing an example of spectral characteristics of abeam splitter.

FIG. 5 is a graph showing an example of a relationship between visiblelight division ratio and sensitivity, dynamic range, and resolution in acase where exposure times of second visible light and first visiblelight are the same.

FIG. 6 is a graph showing an example of the relationship between visiblelight division ratio and sensitivity, dynamic range, and resolution in acase where a ratio of the exposure times of the second visible light andthe first visible light is 10:1.

FIG. 7 is a graph showing an example of the relationship between visiblelight division ratio and sensitivity, dynamic range, and resolution in acase where the ratio of the exposure times of the second visible lightand the first visible light is 100:1.

FIG. 8 is a graph showing an example of the relationship between visiblelight division ratio and sensitivity, dynamic range, and resolution in acase where the ratio of the exposure times of the second visible lightand the first visible light is 1:10.

FIG. 9 is a diagram showing a display example of a visible/IR combinedvideo signal generated by the 3 MOS camera according to the firstembodiment on a monitor.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENT

Hereinafter, embodiments that specifically disclose a 3 MOS cameraaccording to the present disclosure will be described in detail withreference to drawings as appropriate.

However, more detailed description than necessary may be omitted. Forexample, detailed description of a well-known matter and redundantdescription of substantially the same configuration may be omitted. Thisis to prevent the following description from being unnecessarilyredundant and to facilitate understanding by those skilled in the art.The accompanying drawings and the following description are provided forthose skilled in the art to fully understand the present disclosure, andare not intended to limit subject matters described in claims thereby.

FIG. 1A is a block diagram showing an internal configuration example ofa 3 MOS camera 1 according to a first embodiment.

FIG. 1B is a block diagram showing another internal configurationexample of the 3 MOS camera 1 according to the first embodiment. The 3MOS camera 1 includes a lens 11, a spectral prism 13, imaging elements151, 152, and 153, and a video signal processing unit 17. The videosignal processing unit 17 includes camera signal processing units 191,192, and 193, a pixel shifting combination/resolution enhancementprocessing unit 21, and a visible/IR combination processing unit 23. Asshown in FIG. 1B, the 3 MOS camera 1 may include a video signalprocessing unit 17A (refer to FIG. 1B) having a long/short exposurecombination/wide dynamic range processing unit 21A, instead of the videosignal processing unit 17 (refer to FIG. 1A). Although not shown, the 3MOS camera 1 may include both the video signal processing unit 17 (referto FIG. 1A) and the video signal processing unit 17A (refer to FIG. 1B).Each configuration will be described in detail.

The 3 MOS camera 1 is used for a medical observation system in whichexcitation light in a predetermined wavelength band (for example, 760 nmto 800 nm) is emitted to a fluorescent reagent (for example, indocyaninegreen; hereinafter referred to as “ICG”) administered in advance to anobservation part (for example, diseased part) in a subject such as apatient and the observation part that emits fluorescent light on a longwavelength side (for example, 820 to 860 nm) based on the excitationlight is imaged, at the time of surgery or examination, for example. Animage (for example, video of the observation part) captured by the 3 MOScamera 1 is displayed on a monitor MN1 (refer to FIG. 9) and assists auser such as a doctor in executing a medical procedure. The spectralprism 13 will be described as examples used in the medical observationsystem described above. However, the use thereof is not limited tomedical usage and the prism may be used for industrial usage.

Although not shown in FIG. 1, a part of an objective side (in otherwords, tip side) of the 3 MOS camera 1 with respect to the lens 11 isconfigured by a scope that is inserted through the observation part (forexample, diseased part; the same applies hereinafter). This scope is,for example, a main portion of a medical instrument such as a rigidendoscope inserted into the observation part and is a light guide membercapable of guiding light L1 from the observation part to the lens 11.

The lens 11 is attached to the objective side (in other words, tip side)of the spectral prism 13 and collects the light L1 from the observationpart (for example, reflected light at the observation part). Collectedlight L2 is incident on the spectral prism 13.

The spectral prism 13 receives the light L2 from the observation partand splits the light into first visible light V1, a second visible lightV2, and IR light N1. The spectral prism 13 has a configuration having anIR prism 31, visible prisms 32 and 33 (refer to FIG. 2). The firstvisible light V1 is incident on the imaging element 151 disposed so asto face the visible prism 32. The second visible light V2 is incident onthe imaging element 152 disposed so as to face the visible prism 33. TheIR light N1 is incident on the imaging element 153 disposed so as toface the IR prism 31. A detailed structural example of the spectralprism 13 will be described below with reference to FIG. 2.

The imaging element 151 as an example of a second image sensor includes,for example, a charge coupled device (CCD) or a complementary metaloxide semiconductor

(CMOS) in which a plurality of pixels suitable for imaging visible lightare arranged, and an exposure control circuit (not shown) using anelectronic shutter. In the CCD or CMOS constituting the imaging element151, at least a microlens, a color filter, and a light receiving elementare disposed, for example, corresponding to each pixel. The microlenscollects incident light (visible light). The color filter transmitsvisible light of a specific color component (wavelength) transmittedthrough the microlens. The color filter of the imaging element 151 isdisposed in a Bayer array (refer to FIG. 3A) such as red (R), green (G),green (G), and blue (B). The specific color component indicates, forexample, red (R), green (G), and blue (B). The light receiving elementreceives light of the specific color component (wavelength) transmittedthrough the color filter. The imaging element 151 is disposed so as toface the visible prism 32 (refer to FIG. 2). The imaging element 151captures an image based on the first visible light V1 that is incidentfor a first exposure time determined by the exposure control circuitbased on an exposure control signal CSH1 from the camera signalprocessing unit 191. The imaging element 151 generates a video signalV1V of the observation part by imaging and outputs the signal to thevideo signal processing unit 17.

The imaging element 152 as an example of a third image sensor includes,for example, a CCD or a CMOS in which a plurality of pixels suitable forimaging visible light are arranged, and an exposure control circuit (notshown) using an electronic shutter. In the CCD or CMOS constituting theimaging element 152, at least a microlens, a color filter, and a lightreceiving element are arranged, for example, corresponding to eachpixel. The microlens collects incident light (visible light). The colorfilter transmits visible light of a specific color component(wavelength) transmitted through the microlens. The color filter of theimaging element 152 is disposed in the Bayer array (refer to FIG. 3A)such as red (R), green (G), green (G), and blue (B). The specific colorcomponent indicates, for example, red (R), green (G), and blue (B). Thelight receiving element receives light of the specific color component(wavelength) transmitted through the color filter. The imaging element152 is disposed so as to face the visible prism 33 (refer to FIG. 2).The imaging element 152 captures an image based on the second visiblelight V2 that is incident for a second exposure time determined by theexposure control circuit based on an exposure control signal CSH2 fromthe camera signal processing unit 192.

The imaging element 152 generates a video signal V2V of the observationpart by imaging and outputs the signal to the video signal processingunit 17.

The imaging element 153 as an example of a first image sensor includes,for example, a CCD or a CMOS in which a plurality of pixels suitable forimaging IR light are arranged. The imaging element 153 is disposed so asto face the IR prism 31 (refer to FIG. 2). The imaging element 153captures an image based on the incident IR light N1. The imaging element153 generates a video signal N1V of the observation part by imaging andoutputs the signal to the video signal processing unit 17.

The video signal processing unit 17 is configured of a processor such asa digital signal processor (DSP) or a field programmable gate array(FPGA). The camera signal processing units 191 to 193, the pixelshifting combination/resolution enhancement processing unit 21, and thevisible/IR combination processing unit 23 are executed by the processordescribed above.

The camera signal processing unit 191 performs various types of camerasignal processing using the video signal V1V from the imaging element151 to generate a first visible video signal V1VD of the observationpart, and outputs the signal to the pixel shiftingcombination/resolution enhancement processing unit 21 or the long/shortexposure combination/wide dynamic range processing unit 21A. The camerasignal processing unit 191 generates the exposure control signal CSH1for determining the first exposure time of the imaging element 151 andoutputs the signal to the imaging element 151. The imaging element 151controls the exposure time of the first visible light V1 based on theexposure control signal CSH1.

The camera signal processing unit 192 performs various types of camerasignal processing using the video signal V2V from the imaging element152 to generate a second visible video signal V2VD of the observationpart, and outputs the signal to the pixel shiftingcombination/resolution enhancement processing unit 21 or the long/shortexposure combination/wide dynamic range processing unit 21A. Althoughthe details will be described below, brightness (sensitivity) of thefirst visible video signal V1VD and brightness of the second visiblevideo signal V2VD may be substantially the same (including the same) ormay be different. In particular, the closer the brightness (sensitivity)of the first visible video signal V1VD and the brightness of the secondvisible video signal V2VD are to substantially the same (including thesame), the higher an effect of resolution enhancement is. The camerasignal processing unit 192 generates the exposure control signal CSH2for determining the exposure time of the imaging element 152 and outputsthe signal to the imaging element 152.

The imaging element 152 controls the second exposure time of the secondvisible light V2 based on the exposure control signal CSH2. Although thedetails will be described below, the first exposure time and the secondexposure time may be the same (refer to FIG. 5) or may be different(refer to FIGS. 6 to 8), and the same applies hereinafter.

The camera signal processing unit 193 performs various types of camerasignal processing using the video signal N1V from the imaging element153 to generate an IR video signal N1VD of the observation part, andoutputs the signal to the visible/IR combination processing unit 23.

The pixel shifting combination/resolution enhancement processing unit 21receives two video signals (specifically, the first visible video signalV1VD from the camera signal processing unit 191 and the second visiblevideo signal V2VD from the camera signal processing unit 192). Thecloser the brightness of the first visible video signal V1VD and thebrightness of the second visible video signal V2VD are to the same, thehigher the effect of resolution enhancement by the pixel shiftingcombination/resolution enhancement processing unit 21 is.Combination/pixel interpolation processing is performed in considerationof a spatial positional relationship between the first visible videosignal V1VD and the second visible video signal V2VD, and thus it ispossible to generate a high-resolution video signal VVD with highresolution.

The pixel shifting combination/resolution enhancement processing unit 21performs combination processing on the received two input video signals(that is, combination of the first visible video signal V1VD generatedby the camera signal processing unit 191 based on the imaging of theimaging element 151 bonded to the visible prism 32 and the secondvisible video signal V2VD generated by the camera signal processing unit192 based on the imaging of the imaging element 152 bonded to thevisible prism 33) to generate the high-resolution video signal VVD. Withthe combination processing (refer to above) on the received two inputvideo signals, the pixel shifting combination/resolution enhancementprocessing unit 21 can generate the high-resolution video signal VVDhaving higher resolution than the first visible video signal V1VD or thesecond visible video signal V2VD. The pixel shiftingcombination/resolution enhancement processing unit 21 outputs thehigh-resolution video signal VVD to the visible/IR combinationprocessing unit 23. The generation of the high-resolution video signalVVD by the pixel shifting combination/resolution enhancement processingunit 21 will be described below with reference to FIG. 3A.

In the 3 MOS camera 1, the video signal processing unit 17 generates thehigh-resolution video signal VVD by pixel shifting. Therefore, in thespectral prism 13 (refer to FIG. 2), when the imaging element 151 onwhich the first visible light V1 is incident and the imaging element 152on which the second visible light V2 is incident are respectively bondedto the corresponding visible prisms 32 and 33, it is necessary tooptically shift positions of the imaging element 151 and the imagingelement 152 by substantially one pixel (for example, in the horizontalor vertical direction, or in both directions) to perform the bonding(refer to FIG. 3A). Accordingly, the pixel shiftingcombination/resolution enhancement processing unit 21 can generate thehigh-resolution video signal VVD by the pixel shifting based on theimaging of the imaging elements 151 and 152 which are disposed in anoptically shifted manner by substantially one pixel (refer to above).The substantially one pixel includes one pixel, may not be exactly onepixel, and may include, for example, a distance deviation of one pixelplus or minus 0.25 pixels. The closer an amount of pixel shifting is toone pixel, the higher the effect of resolution enhancement by the pixelshifting combination/resolution enhancement processing unit 21 is.

The long/short exposure combination/wide dynamic range processing unit21A receives and superimposes the two video signals having differentbrightness (sensitivity) (specifically, the first visible video signalV1VD from the camera signal processing unit 191 and the second visiblevideo signal V2VD from the camera signal processing unit 192) forcombining the signals to generate a wide dynamic range video signalVVDA. The long/short exposure combination/wide dynamic range processingunit 21A superimposes and combines the two video signals havingdifferent brightness (sensitivity) and thus can generate the widedynamic range video signal VVDA with an apparently wider dynamic rangethan the first visible video signal V1VD or the second visible videosignal V2VD. The long/short exposure combination/wide dynamic rangeprocessing unit 21A outputs the wide dynamic range video signal VVDA tothe visible/IR combination processing unit 23.

The visible/IR combination processing unit 23 receives and superimposesthe high-resolution video signal VVD from the pixel shiftingcombination/resolution enhancement processing unit 21 and the IR videosignal N1VD from the camera signal processing unit 193 for combining thesignals to generated a visible/IR combined video signal IMVVD. With thevisible/IR combined video signal IMVVD, the resolution is enhanced bythe combination processing after the pixel shifting. Therefore, a statearound the observation part (for example, surgical field) becomesvisually clear and a state of the diseased part can be clarified indetail by the fluorescent light emission of the fluorescent reagent suchas ICG (refer to FIG. 9). The visible/IR combination processing unit 23may output the visible/IR combined video signal IMVVD to the monitor MN1or send the signal to a recording device (not shown) for accumulation.

The monitor MN1 constitutes, for example, an image console (not shown)disposed in a surgery room at the time of surgery or examination, anddisplays the visible/IR combined video signal IMVVD of the observationpart generated by the 3 MOS camera 1. Accordingly, the user such asdoctor can visually recognize the visible/IR combined video signal IMVVDdisplayed on the monitor MN1 to grasp in detail the part that emitsfluorescent light in the observation part. The recording device is arecorder capable of recording data of the visible/IR combined videosignal IMVVD generated by the 3 MOS camera 1, for example.

FIG. 2 is a diagram showing a structural example of the spectral prism13 shown in FIG. 1. Hereinafter, the structural example of the spectralprism 13 shown in FIG. 1 will be mainly described with reference to FIG.2. The spectral prism 13 includes the IR prism 31 (an example of a firstprism), the visible prism 32 (an example of a second prism), and thevisible prism 33 (an example of a third prism). The IR prism 31, thevisible prism 32, and the visible prism 33 are sequentially assembled inan optical axis direction of the light L2 collected by the lens 11.

The IR prism 31 as an example of the first prism includes an incidentsurface 31 a on which the light L2 is incident, a reflection surface 31b on which a dichroic mirror DYM1 that reflects the IR light of thelight L2 is formed, and an emission surface 31 c from which the IR lightis emitted. The dichroic mirror DYM1 (an example of first reflectionfilm) is formed on the reflection surface 31 b by vapor deposition orthe like, reflects the IR light (for example, IR light in the wavelengthband of 800 nm or more) of the light L2, and transmits light (forexample, light of about 400 nm to 800 nm) other than the IR light of thelight L2 (refer to FIG. 4A). Specifically, the IR light (refer to above)of the light L2 incident on the incident surface 31 a of the IR prism 31is reflected by the reflection surface 31 b. This IR light is reflectedby the reflection surface 31 b, is then totally reflected by theincident surface 31 a of the IR prism 31, and is incident on the imagingelement 153 through the emission surface 31 c.

FIG. 4A is a graph showing an example of spectral characteristics of thedichroic mirror DYM1. The horizontal axis of FIG. 4A indicateswavelength [nm: nanometer (the same applies hereinafter)], and thevertical axis indicates reflectance or transmittance. A characteristicTP1 indicates the transmittance of the dichroic mirror DYM1. Accordingto the characteristic TP1, the dichroic mirror DYM1 can transmit thelight of about 400 nm to 800 nm. A characteristic RF1 indicates thereflectance of the dichroic mirror DYM1. According to the characteristicRF1, the dichroic mirror DYM1 can reflect the IR light of 800 nm ormore. Therefore, all the IR light having a light amount indicated by anarea AR1 (in other words, the IR light of the light L2) can be incidenton the imaging element 153.

The visible prism 32 as an example of the second prism includes anincident surface 32 a on which the light (an example of firsttransmitted light) transmitted through the dichroic mirror DYM1 isincident, a reflection surface 32 b on which a beam splitter BSP1 forreflecting a partial light amount of the transmitted light(specifically, visible light) is formed, and an emission surface 32 cfrom which reflected visible light of the partial light amount isemitted. The beam splitter BSP1 (an example of second reflection film)is formed on the reflection surface 32 b by vapor deposition or thelike, reflects visible light having a partial (for example, around A %of the light incident on the incident surface 32 a; A is a predeterminedreal number, for example, 50) light amount of the visible light incidenton the incident surface 32 a, and transmits visible light having aremaining (100-A)% (for example, around 50% of the light incident on theincident surface 32 a) light amount thereof (refer to FIG. 4B).Specifically, the visible light having the partial (for example, 50%)light amount of the visible light incident on the incident surface 32 aof the visible prism 32 is reflected by the reflection surface 32 b.This part of the visible light is reflected by the reflection surface 32b, is then totally reflected by the incident surface 32 a of the visibleprism 32, and is incident on the imaging element 151 through theemission surface 32 c. In the spectral prism 13 shown in FIG. 1, a ratioof visible light reflected by the beam splitter BSP1 is not limited to50% and may be in a range of 30% to 50%, for example.

The visible prism 33 as an example of the third prism has an incidentsurface 33 a on which the visible light having the remaining lightamount transmitted through the beam splitter BSP1 is incident and anemission surface 33 c from which the visible light having the remaininglight amount is emitted. Specifically, the visible light having theremaining light amount transmitted through the beam splitter BSP1 isincident on the visible prism 33, is emitted as it is, and is incidenton the imaging element 152 (refer to FIG. 4B).

FIG. 4B is a graph showing an example of spectral characteristics of thebeam splitter BSP1. The horizontal axis of FIG. 4B indicates wavelength[nm], and the vertical axis indicates reflectance or transmittance. Acharacteristic TP2 indicates transmittance and reflectance (about 50% at400 nm to 800 nm) of the beam splitter BSP1 in the spectral prism 13shown in FIG. 2. With the characteristic TP2, the beam splitter BSP1 asan example of the second reflection film can reflect light having alight amount of about 50% (mainly visible light) of the light of about400 nm to 800 nm and can transmit light having a remaining light amountof about 50% (mainly visible light) thereof. Therefore, visible lighthaving a light amount indicated by an area AR2 (for example, visiblelight having light amount of about 50%) can be incident on the imagingelement 151. The visible light having the light amount indicated by thearea AR2 (for example, visible light having light amount of about 50%)can be incident on the imaging element 152.

Next, the arrangement of color filters BYR1 and BYR2 of the imagingelements 151 and 152 will be described with reference to FIG. 3A. FIG.3A is a diagram showing an arrangement example of the color filters BYR1and BYR2 of the imaging elements 151 and 152. The color filter BYR1 is acolor filter constituting the imaging element 151 and is disposed in theBayer array consisting of the color filters of red (R), green (G), green(G), and blue (B) in any four adjacent pixels in the horizontal andvertical directions, for example. In the Bayer array, more green (G) isdisposed than red (R) and blue (B) in any four pixels.

This is because human vision is known to react most sensitively to green(G). Similarly, the color filter BYR2 is a color filter constituting theimaging element 152 and is disposed in the Bayer array consisting of thecolor filters of red (R), green (G), green (G), and blue (B) in any fouradjacent pixels in the horizontal and vertical directions, for example.

As shown in FIG. 3A, the imaging elements 151 and 152 are disposed withan offset of one pixel, and thus the color filters BYR1 and BYR2 aredisposed so as to be offset by one pixel. Although FIG. 3A shows anexample in which the offset of one pixel is added, the color filtersBYR1 and BYR2 may be disposed with an offset of substantially one pixel(refer to above). Therefore, with the pixel shifting of the offset ofsubstantially one pixel (refer to above), the green (G) pixel of oneBayer array (for example, the color filter BYR1) is disposed on the blue(B) pixel or the red (R) pixel of the other Bayer array (for example,the color filter BYR2). In other words, the green (G) color filter isdisposed for all pixels. Accordingly, the pixel shiftingcombination/resolution enhancement processing unit 21 that receives thefirst visible video signal V1VD and the second visible video signal V2VDcan generate the high-resolution video signal VVD having high resolutionas compared with a video signal in a case where the pixel shifting bysubstantially one pixel is not performed, by selectively using lighttransmitted through the green (G) color filter, which has the highestratio of contributing to resolution of a luminance signal in each pixel,of the color filters BYR1 and BYR2 of the Bayer array stacked in twolayers (refer to FIG. 3A).

A problem in a case where the color filters of the imaging elements 151and 152 are disposed with a pixel shifting offset by a half pixel willbe described with reference to FIG. 3B. FIG. 3B is an explanatorydiagram of a problem in a case where the color filters BYR1 and BYR2 ofthe imaging elements 151 and 152 are configured in the Bayer array anddisposed with the half pixel shifting. The horizontal axis and thevertical axis of FIG. 3B are both frequencies, where fs indicatessampling frequency and fs/2 indicates Nyquist frequency.

In a case where the color filters of the imaging elements 151 and 152are stacked and disposed with the pixel shifting offset by the halfpixel, it is found that false color or moire, which is not present inthe subject, is detected near the Nyquist frequency (fs/2) as shown inFIG. 3B. When such false color or moire is detected, the image qualityof the color video signal deteriorates. On the other hand, in order tosolve such a problem, in the first embodiment, the color filters BYR1and BYR2 are disposed with the optical offset of one pixel as shown inFIG. 3A. Accordingly, in the high-resolution video signal VVD generatedby the pixel shifting combination/resolution enhancement processing unit21, there is no detection of the false color or moire as shown in FIG.3B near the Nyquist frequency (fs/2) and the image quality is accuratelyenhanced.

FIG. 5 is a graph showing an example of a relationship between visiblelight division ratio and sensitivity GAN1, dynamic range DRG1, andresolution RSO1 in a case where exposure times of the second visiblelight V2 and the first visible light V1 are the same. The horizontalaxis of FIG. 5 is the visible light division ratio. In other words, thevisible light division ratio is a ratio at which the beam splitter BSP1reflects the visible light transmitted through the dichroic mirror DYM1.For example, in a case where the visible light division ratio is 10%(that is, 90:10), the beam splitter BSP1 reflects the visible light of10% of the visible light transmitted through the dichroic mirror DYM1and transmits the visible light of 90% thereof. That is, the ratio lightamount of the second visible light

V2:light amount of the first visible light V1 is 90:10. Another visiblelight division ratio can be considered in the same manner as thespecific example described above. The vertical axis of FIG. 5 shows thesensitivity GAN1, the dynamic range DRG1, and the resolution RSO1 of thehigh-resolution video signal VVD generated by the video signalprocessing unit 17.

FIG. 5 shows an example in which the exposure times for the imagingelements 152 and 151 by the electronic shutter are controlled to be thesame. Therefore, it is considered that the sensitivity GAN1 transitionsaccording to a characteristic (for example, a linear function) that thesensitivity is the maximum as the visible light division ratio issmaller (for example, the maximum (100%) and the brightest when theratio is 0%) and the sensitivity is the minimum (for example, thedarkest at 50%) when the ratio is 50%. This is because the sensitivityis determined by the brightness of the brighter second visible light V2of the brightness of the first visible video signal V1VD based on thefirst visible light V1 and the brightness of the second visible videosignal V2VD based on the second visible light V2.

It is considered that the dynamic range DRG1 transitions according to acharacteristic that the dynamic range increases similarly as the visiblelight division ratio is smaller in a range larger than zero (forexample, about +80 dB when the ratio is 0.01%) and the dynamic range isthe minimum (for example, 0 dB) when the ratio is 50%. This is because adifference between a dark portion and a bright portion tends to widen asthe visible light division ratio is smaller in the high-resolution videosignal VVD.

It is considered that the resolution RSO1 transitions according to acharacteristic that the resolution is the minimum contrarily as thevisible light division ratio is smaller (for example, the maximum of 1time when the ratio is 0%) and the resolution is the maximum (forexample, 1.1 times) when the ratio is 50%. This is because a differencein pixel value between adjacent pixels is small as the visible lightdivision ratio is larger and thus it is easy to realize high resolutionby pixel shifting.

FIG. 6 is a graph showing an example of a relationship between visiblelight division ratio and sensitivity GAN2, dynamic range DRG2, andresolution RSO2 in a case where a ratio of the exposure times of thesecond visible light V2 and the first visible light V1 is 10:1. Thehorizontal axis of FIG. 6 is the visible light division ratio, anddescription thereof will be omitted since the description is the same asthat in FIG. 5. The vertical axis of FIG. 6 shows the sensitivity GAN2,the dynamic range DRG2, and the resolution RSO2 of the high-resolutionvideo signal VVD generated by the video signal processing unit 17.

FIG. 6 shows an example in which a difference is provided such that aratio of the exposure times for the imaging elements 152 and 151 by theelectronic shutter is 10:1. It is considered, as in the case of thesensitivity GAN1 shown in FIG. 5, that the sensitivity GAN2 transitionsaccording to a characteristic (for example, a linear function) that thesensitivity is the maximum as the visible light division ratio issmaller (for example, the maximum (100%) and the brightest when theratio is 0%) and the sensitivity is the minimum (for example, thedarkest at 50%) when the ratio is 50%. This is because a brightnessratio of the second visible video signal V2VD and the first visiblevideo signal V1VD is obtained by multiplying the ratio of the exposuretimes for the imaging elements 152 and 151 of 10:1 by a light amountratio of the second visible light V2 and the first visible light V1, andthe sensitivity is determined by the brightness of the brighter secondvisible video signal V2VD of the second visible video signal V2VD andthe first visible video signal V1VD.

When a difference is provided such that the ratio of the exposure timesfor the imaging elements 152 and 151 is, for example, 10:1 as comparedwith when the exposure time thereof is the same, it is considered thatthe difference between the bright portion and the dark portion is likelyto appear further clearly and thus it is possible to gain more dynamicrange, in the high-resolution video signal VVD. Therefore, it isconsidered that the dynamic range DRG2 transitions according to acharacteristic that the dynamic range increases similarly as the visiblelight division ratio is smaller in a range larger than zero (forexample, about +80 dB when the ratio is 0.1%) and the dynamic range isthe minimum (for example, +20 dB) when the ratio is 50%. That is, it ispossible to gain +20 dB even with a minimum value in the example of FIG.6.

When the difference is provided such that the ratio of the exposuretimes for the imaging elements 152 and 151 by the electronic shutter is10:1, it is considered that the ratio light amount of light incident onthe imaging element 152:light amount of light incident on the imagingelement 151 =100:1 in a case where the visible light division ratio is10% (the ratio second visible light V2:first visible light V1 =90:10).That is, the dark portion is hardly projected by the first visible lightV1 and the bright portion is hardly projected by the second visiblelight V2, and thus it can be considered that it is almost difficult togain a resolution when two video signals are superimposed. Therefore, itis considered that the resolution RSO2 transitions over small values(for example, the minimum of 1 time at 0% and about 1.02 times at 50%)regardless of the visible light division ratio.

FIG. 7 is a graph showing an example of a relationship between visiblelight division ratio and sensitivity GAN2, dynamic range DRG3, andresolution RSO3 in a case where the ratio of the exposure times of thesecond visible light V2 and the first visible light V1 is 100:1. Thehorizontal axis of FIG. 7 is the visible light division ratio, anddescription thereof will be omitted since the description is the same asthat in FIG. 5. The vertical axis of FIG. 7 shows the sensitivity GAN2,the dynamic range DRG3, and the resolution RSO3 of the high-resolutionvideo signal VVD generated by the video signal processing unit 17.

FIG. 7 shows an example in which a considerable difference is providedsuch that the ratio of the exposure times for the imaging elements 152and 151 by the electronic shutter is 100:1. It is considered, as in thecase of the sensitivity GAN2 shown in FIG. 6, that the sensitivity GAN2transitions according to a characteristic (for example, a linearfunction) that the sensitivity is the maximum as the visible lightdivision ratio is smaller (for example, the maximum (100%) and thebrightest when the ratio is 0%) and the sensitivity is the minimum (forexample, the darkest at 50%) when the ratio is 50%. This is because abrightness ratio of the second visible video signal V2VD and the firstvisible video signal V1VD is obtained by multiplying the ratio of theexposure times for the imaging elements 152 and 151 of 100:1 by a lightamount ratio of the second visible light V2 and the first visible lightV1, and the sensitivity is determined by the brightness of the brightersecond visible video signal V2VD of the second visible video signal V2VDand the first visible video signal V1VD.

When a considerable difference is provided such that the ratio of theexposure times for the imaging elements 152 and 151 is, for example,100:1 as compared with when the exposure time thereof is the same, it isconsidered that the difference between the bright portion and the darkportion is likely to appear furthermore clearly and thus it is possibleto gain more dynamic range, in the high-resolution video signal VVD.Therefore, it is considered that the dynamic range DRG3 transitionsaccording to a characteristic that the dynamic range increases similarlyas the visible light division ratio is smaller in a range larger thanzero (for example, about +80 dB when the ratio is 1%) and the dynamicrange is the minimum (for example, +40 dB) when the ratio is 50%. Thatis, it is possible to gain +40 dB even with a minimum value in theexample of FIG. 7.

When the difference is provided such that the ratio of the exposuretimes for the imaging elements 152 and 151 by the electronic shutter is100:1, it is considered that the ratio light amount of light incident onthe imaging element 152:light amount of light incident on the imagingelement 151 =1000:1 in the case where the visible light division ratiois 10% (the ratio second visible light V2:first visible light V1=90:10). That is, the dark portion is hardly projected since the secondvisible light V2 is too bright and the bright portion is hardlyprojected since the first visible light V1 is too dark, and thus it canbe considered that it is almost difficult to gain a resolution when twovideo signals are superimposed as compared with the example of FIG. 6.Therefore, it is considered that the resolution RSO3 transitions oversmall values (for example, the minimum of 1 time at 0% and about 1.001times at 50%) regardless of the visible light division ratio.

FIG. 8 is a graph showing an example of a relationship between visiblelight division ratio and sensitivity GAN3, dynamic range DRG4, andresolution RSO4 in a case where the ratio of the exposure times of thesecond visible light V2 and the first visible light V1 is 1:10. Thehorizontal axis of FIG. 8 is the visible light division ratio, anddescription thereof will be omitted since the description is the same asthat in FIG. 5. The vertical axis of FIG. 8 shows the sensitivity GAN3,the dynamic range DRG4, and the resolution RSO4 of the high-resolutionvideo signal VVD generated by the video signal processing unit 17.

FIG. 8 shows an example in which a difference is provided such that theratio of the exposure times for the imaging elements 152 and 151 by theelectronic shutter is 1:10.

Contrary to the example of FIG. 6, when the difference is provided suchthat the ratio of the exposure times for the imaging elements 152 and151 is, for example, 1:10, it is considered that the light amount oflight incident on the imaging element 152 and the light amount of lightincident on the imaging element 151 are substantially equal due tocancellation of the visible light division ratio and the exposure timeratio in the case where the visible light division ratio is 10% (secondvisible light V2:first visible light V1 =90:10), for example. Therefore,it is considered that the sensitivity GAN3 transitions according to acharacteristic that the sensitivity transitions substantially constantso as to be the minimum when the visible light division ratio is from 0%to 10% (in other words, in a case where light amounts incident on theimaging elements 152 and 151 do not change much) and the sensitivityincreases monotonically in a linear function until the visible lightdivision ratio is larger than 10% and reaches 50%. For example, thebrightness is the maximum (50%, that is, −6 dB) when the visible lightdivision ratio is 50%. This is because a brightness ratio of the secondvisible video signal V2VD and the first visible video signal V1VD isobtained by multiplying the ratio of the exposure times for the imagingelements 152 and 151 of 1:10 by a light amount ratio of the secondvisible light V2 and the first visible light V1, and the sensitivity isdetermined by the brightness of a brighter video signal of the secondvisible video signal V2VD and the first visible video signal V1VD.

When a difference is provided such that the ratio of the exposure timesfor the imaging elements 152 and 151 is, for example, 1:10 as comparedwith when the exposure time thereof is the same, it is considered thatthe difference in brightness is easier to obtain as the visible lightdivision ratio is smaller in a range larger than 0%, but the differencebetween the bright portion and the dark portion is less likely to appearas the visible light division ratio is higher and thus it is difficultto gain more dynamic range, in the high-resolution video signal VVD.Therefore, the dynamic range DRG4 increases as the visible lightdivision ratio is smaller in a range larger than 0% (for example, about+80 dB at 0.001%). However, when the visible light division ratio is10%, the brightness of the second visible video signal V2VD and thebrightness of the first visible video signal V1VD are substantiallyequal due to the cancellation of the visible light division ratio andthe ratio of the exposure times for the imaging elements 152 and 151 of1:10 and the dynamic range DRG4 is the minimum. When the visible lightdivision ratio exceeds 10%, the brightness of the second visible videosignal V2VD is different again from the brightness of the first visiblevideo signal V1VD and the dynamic range DRG4 is large. When the visiblelight division ratio is 50%, the ratio of the brightness of the secondvisible video signal V2VD and the brightness of the first visible videosignal V1VD is 1:10 by the multiplication of the ratio of the exposuretimes for the imaging elements 152 and 151 of 1:10 and the dynamic rangeis +20 dB.

When the difference is provided such that the ratio of the exposuretimes for the imaging elements 152 and 151 by the electronic shutter is1:10, it is considered that the light amount of light incident on theimaging element 152 and the light amount of light incident on theimaging element 151 are substantially equal in the case where thevisible light division ratio is 10% (the ratio second visible lightV2:first visible light V1 =90:10), for example (refer to above). Thatis, when the cancellation of the visible light division ratio and theexposure time ratio (1:10) occurs (for example, when the visible lightdivision ratio is 10%), the first visible video signal V1VD based on thefirst visible light V1 and the second visible video signal V2VD based onthe second visible light V2 have the same brightness. Therefore, it isconsidered that the resolution RSO4 transitions according to acharacteristic that the resolution is the maximum and the resolutiondecreases from the maximum value at a visible light division ratio atwhich the cancellation is less likely to occur.

FIG. 9 is a diagram showing a display example of the visible/IR combinedvideo signal IMVVD generated by the 3 MOS camera 1 according to thefirst embodiment on the monitor MN1. The visible/IR combined videosignal IMVVD shown in FIG. 9 is generated based on imaging at theobservation part (for example, around liver and pancreas) of the patientwho is the subject and is displayed on the monitor MN1. In FIG. 9, thefluorescent reagent of ICG, which is administered in advance to thediseased part in a body of the patient before surgery or examination,emits light, and a place that emits the light (for example, diseasedpart FL1) is shown so as to be known in the visible/IR combined videosignal IMVVD. The high-resolution video signal VVD having the highresolution is generated by the pixel shifting combination/resolutionenhancement processing unit 21. Therefore, a clear video of the surgicalfield such as an observation target can be obtained with the visible/IRcombined video signal IMVVD. In this manner, the 3 MOS camera 1 cangenerate the visible/IR combined video signal IMVVD, which allows theuser such as doctor to grasp the details of the observation part withhigh image quality and to easily specify a position of the diseasedpart, and display the signal on the monitor MN1, at the time of surgeryor examination, for example.

As described above, the 3 MOS camera 1 according to the first embodimentis provided with the first prism (for example, IR prism 31) that causesthe imaging element 153 to receive the IR light of the light L2 from theobservation part (for example, diseased part in the subject), the secondprism (for example, visible prism 32) that reflects the visible light ofA % of the light L2 from the observation part (for example, diseasedpart in the subject) and causes the imaging element 151 to receive theremaining (100-A)% thereof, and the third prism (for example, visibleprism 33) that causes the imaging element 152 to receive the remainingvisible light of (100-A)% thereof. The 3 MOS camera 1 is provided withthe video signal processing unit 17 that combines the color video signalbased on the imaging outputs of the imaging element 151 and the imagingelement 152, which are respectively bonded to the positions opticallyshifted by substantially one pixel, and the IR video signal based on theimaging output of the imaging element 153, and outputs the combinedsignal to the monitor MN1.

Accordingly, the 3 MOS camera 1 can separate (split), by the spectralprism 13, the IR light specialized in a fluorescent region of thefluorescent reagent of the light from the observation part (for example,diseased part) to which the fluorescent reagent (for example, ICG) isadministered in advance in the subject such as patient at the time ofsurgery or examination, for example.

The 3 MOS camera 1 can generate an RGB color video signal having highresolution based on the imaging outputs of the imaging elements 151 and152, which are optically shifted by substantially one pixel, obtained byreflecting the part of the visible light of the light from theobservation part and transmitting the remaining visible light thereof onthe beam splitter BSP1. The 3 MOS camera 1 can generate an RGB colorvideo signal with an expanded dynamic range by combining the imagingoutputs of the imaging elements 151 and 152. The 3 MOS camera 1 cangenerate and output clearer fluorescence images in both the IR light andthe visible light and thus achieve both the generation of a clearerfluorescence video of the observation part to which the fluorescentreagent is administered and the resolution enhancement of the colorimage of the observation part to assist the doctor or the like in easilygrasping the diseased part.

The first reflection film (for example, dichroic mirror DYM1) thatreflects the IR light is formed on the first prism. The secondreflection film (for example, beam splitter BSP1) that reflects thevisible light of A % of the visible light transmitted through the firstreflection film and transmits the visible light of (100-A)% thereof isformed on the second prism. The visible light of (100-A)% that transmitsthrough the second reflection film is incident on the third prism. Thedichroic mirror DYM1 first splits the IR light of the light from theobservation part (for example, diseased part), and the visible lighttransmitted through the dichroic mirror DYM1 is split by the beamsplitter BSP1. Therefore, it is possible to improve the efficiency ofthe splitting in the dichroic mirror DYM1 and the beam splitter BSP1.

A value of A % and a value of the remaining (100-A)% are substantiallyequal. The A value becomes substantially 50, and light having equalbrightness is incident on each of the color filters BYR1 and BYR2, whichare optically shifted by substantially one pixel. Therefore, the 3 MOScamera 1 can effectively generate the highest resolution RGB color videosignal.

The color filter BYR1 having red (R), green (G), and blue (B) of theimaging element 151 and the color filter BYR2 having red (R), green (G),and blue (B) of the imaging element 152 are disposed such that the green(G) color filter is located in each pixel. The video signal processingunit 17 selects a pixel value based on the green (G) color filterdisposed so as to be located in each pixel and mainly uses the selectedpixel value to generate the luminance signal among the color videosignals. Accordingly, the video signal processing unit 17 can generatethe high-resolution video signal VVD having high resolution as comparedwith the video signal in a case where the pixel shifting bysubstantially one pixel is not performed, by selectively using lighttransmitted through the green (G) color filter, which has the highestratio of contributing to resolution of a luminance signal in each pixel,of the color filters BYR1 and BYR2 of the Bayer array stacked in twolayers (refer to FIG. 3A).

This is based on the fact that the green (G) color filter is known tohave the highest proportion of contributing to the resolution of theluminance signal since human vision is most sensitive to green (G).

The imaging element 152 is disposed so as to be optically shifted by onepixel in at least one of the horizontal direction or the verticaldirection with respect to the imaging element 151. Accordingly, thevideo signal processing unit can generate the high-resolution videosignal VVD by the pixel shifting based on the imaging of the imagingelements 151 and 152 which are disposed in an optically shifted mannerby substantially one pixel (refer to above).

The 3 MOS camera 1 controls the ratio of the exposure times of theimaging elements 151 and 152 to be the same or different. Accordingly,the 3 MOS camera 1 can generate high-quality video signals thatadaptively realize sensitivity, dynamic range, and resolution fitted tothe preference of the user according to the ratio of the exposure timesof the imaging elements 151 and 152 and the reflectance of the visiblelight by the beam splitter BSP1 (refer to FIGS. 5 to 8).

Although various embodiments are described with reference to thedrawings, it goes without saying that the present disclosure is notlimited to such examples. It is obvious to those skilled in the art thatvarious modification examples, change examples, substitution examples,addition examples, deletion examples, and equivalent examples can beconceived within the scope of the claims. Of course, it is understoodthat the various examples belong to the technical scope of the presentdisclosure. Further, the respective constituent elements in the variousembodiments described above may be randomly combined in the scope of notdeparting from the spirit of the invention.

For example, the IR prism 31 is illustrated as an example of the firstprism in the first embodiment described above, but the first prism maynot be limited to the IR prism 31. For example, in a case where thefirst prism is not a visible prism that reflects the visible light, thefirst prism may be a prism that reflects the IR light and light inanother wavelength band (for example, wavelength band of ultravioletray) other than the visible light of the light L2. Accordingly, insteadof the IR video signal, a video obtained by combining, for example, avideo signal based on imaging of the ultraviolet ray and an RGB colorvideo signal with enhanced resolution and expanded dynamic range can beoutput to the monitor MN1 or the like.

In the spectral prism 13 shown in FIG. 2, an example in which the IRprism 31 is disposed most on the objective side has been described, butthe IR prism 31 may not be disposed on the most objective side. Forexample, the IR prism 31 may be disposed at any of the positions of thevisible prisms 32 and 33. With the bonding of the imaging elements 151and 152 to the visible prisms 32 and 33 with the optical shift ofsubstantially one pixel (refer to above), it is possible to obtain thesame effect as that of the 3 MOS camera 1 according to the firstembodiment described above regardless of the position of the IR prism 31on the spectral prism 13.

The present disclosure is useful as the 3 MOS camera that achieves boththe generation of the clearer fluorescence video of the observation partto which the fluorescent reagent is administered and the resolutionenhancement of the color image of the observation part to assist thedoctor or the like in easily grasping the diseased part.

The present application is based upon Japanese Patent Application(Patent Application No. 2020-131042 filed on Jul. 31, 2020), the contentof which is incorporated herein by reference.

1. A 3 MOS camera comprising: a first prism that causes a first imagesensor to receive IR light of light from an observation part; a secondprism that causes a second image sensor to receive A % of the visiblelight of the light from the observation part, wherein A is apredetermined real number, and wherein the visible light from theobservation part not received by the second image sensor is denoted asremaining visible light comprising (100-A)% of the visible light of thelight from the observation part; a third prism that causes a third imagesensor to receive the remaining visible light comprising (100-A)% of thevisible light of the light from the observation part; and a video signalprocessor that combines a color video signal based on imaging outputs ofthe second image sensor and the third image sensor and an IR videosignal based on an imaging output of the first image sensor to produce acombined signal, and outputs the combined signal to a monitor, whereinthe second image sensor and the third image sensor are respectivelybonded to positions optically shifted by substantially one pixel.
 2. The3 MOS camera according to claim 1, further comprising: a firstreflection film formed on the first prism that reflects the IR light ofthe light from the observation part, and transmits the visible light ofthe light from the observation part therethrough; a second reflectionfilm formed on the second prism that reflects the A % of the visiblelight of the light from the observation part transmitted through thefirst reflection film, and transmits the remaining visible lightcomprising (100-A)% of the visible light of the light from theobservation part transmitted through the first reflection film; andwherein the remaining visible light comprising (100-A)% of the visiblelight of the light from the observation part transmitted through thesecond reflection film is incident on the third prism.
 3. The 3 MOScamera according to claim 1, wherein a value of A % and a value of(100-A)% are substantially equal.
 4. The 3 MOS camera according to claim1, wherein the second image sensor comprises a color filter comprisingred, green, and blue filters, the third image sensor comprises a colorfilter comprising red, green, and blue filters, the green color filterof the color filter of the second image sensor is located in each pixelof the second image sensor, the green color filter of the color filterof the third image sensor is located in each pixel of the third imagesensor, and the video signal processor selects a pixel value based onthe green color filters in each pixel to generate the color videosignal.
 5. The 3 MOS camera according to claim 4, wherein the thirdimage sensor is disposed so as to be optically shifted by one pixel inat least one of a horizontal direction and a vertical direction withrespect to the second image sensor.